Skip to content

CSHARP-5614: Fix deserialization of primitive arrays on Big Endian systems #1683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 12, 2025
Merged

CSHARP-5614: Fix deserialization of primitive arrays on Big Endian systems #1683

merged 3 commits into from
Jun 12, 2025

Conversation

medhatiwari
Copy link
Contributor

Description

This PR fixes deserialization issues in PrimitivesArrayReader.cs which caused 51 test failures in the MongoDB.Bson.Tests suite on Big Endian systems (e.g., IBM s390x).

Problem

The deserialization logic relied on BitConverter.ToXxx(...) which assumes Little Endian memory layout. On Big Endian architectures, this results in incorrect values for arrays of primitive types such as Int64, Single, Double, and Decimal, leading to test failures.

Fix

  • Replaced unsafe BitConverter usages with BinaryPrimitives.ReadXxxLittleEndian(...) for endian-safe reads.

  • Covered conversions for:

      -- float, double (via BitConverter.Int64BitsToDouble(...))

  -- decimal (via Decimal128.FromIEEEBits(...))

  -- char (casted from ushort)

  -- All numeric types (e.g. int, long, ushort, etc.)

Results:

51 test failures resolved.

Example fixed tests:

MemorySerializerTests.ReadonlyMemory_should_roundtrip_special_values<Int64>

MemorySerializerTests.Memory_should_roundtrip_special_values<Single>

BinaryVectorSerializerTests.BinaryVectors_should_roundtrip<float>

Pre-requisites:

  • s390x system (e.g., IBM Z, RHEL on s390x)

  • .NET 8.0 SDK installed

cc: @giritrivedi

@medhatiwari
fix(bson): correct deserialization of primitive arrays using BinaryPr…
89a8e71 
…imitives for Big Endian support

Signed-off-by: Medha Tiwari <[email protected]>
@medhatiwari medhatiwari requested a review from a team as a code owner May 6, 2025 06:14
@medhatiwari medhatiwari requested review from rstam and removed request for a team May 6, 2025 06:14
@medhatiwari
Copy link
Contributor Author

hi @rstam , Can you please review this?

@rstam
Copy link
Contributor

rstam commented May 7, 2025

We will discuss this at our next triage meeting. Not sure whether we intend to support Big Endian systems or not.

@medhatiwari
Copy link
Contributor Author

Hi @rstam
Just following up to check if there were any updates or decisions made regarding Big Endian support in the triage meeting?

@medhatiwari
Copy link
Contributor Author

medhatiwari commented Jun 9, 2025 •
edited
Loading

@rstam

In our other PR (#1682), Boris pointed out that:

BinaryPrimitives.ReadSingleBigEndian is not available on older TFMs, i.e. net472 so this won’t compile
#1682 (comment)

We added compatibility shims in BinaryPrimitivesCompat for ReadSingleLittleEndian/WriteSingleLittleEndian.
Here in src/MongoDB.Bson/Serialization/Serializers/PrimitivesArrayReader.cs we’re calling:

BinaryPrimitives.ReadInt32LittleEndian
BinaryPrimitives.ReadUInt32LittleEndian
BinaryPrimitives.ReadInt64LittleEndian
BinaryPrimitives.ReadUInt64LittleEndian

Should we add compatibility shims for these missing Read*LittleEndian APIs in BinaryPrimitivesCompat, just as we did for ReadSingleLittleEndian/WriteSingleLittleEndian? I can also add corresponding unit tests for each shim once they’re in place.

cc: @BorisDog

@BorisDog BorisDog requested review from BorisDog and removed request for rstam June 9, 2025 18:18
BorisDog
BorisDog requested changes Jun 10, 2025
View reviewed changes
Copy link
Contributor

@BorisDog BorisDog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. Few asks/questions:

  1. Could you please file a separate Jira ticket for each PR, describing the issue? And also update the PR name according to our conventions (see the closed PRs for this).

  2. Did you observe any failing failing tests without this fix?

src/MongoDB.Bson/Serialization/Serializers/PrimitivesArrayReader.cs Outdated
@@ -77,6 +78,7 @@ private static T[] ReadBsonArray<T>(

var result = new List<T>();


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for empty lines changes (extra line here, and removed lines below)

src/MongoDB.Bson/Serialization/Serializers/PrimitivesArrayReader.cs Outdated
@@ -95,85 +97,81 @@ private static T[] ReadBsonArray<T>(
{
case ConversionType.DoubleToSingle:
{
var v = (float)BitConverter.ToDouble(bytes, index);

var v = (float)BitConverter.Int64BitsToDouble(BinaryPrimitives.ReadInt64LittleEndian(bytes.AsSpan(index)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably could just use BinaryPrimitivesCompat.ReadDoubleLittleEndian ?

src/MongoDB.Bson/Serialization/Serializers/PrimitivesArrayReader.cs Outdated
value = Unsafe.As<float, T>(ref v);
break;
}
case ConversionType.DoubleToDouble:
{
var v = BitConverter.ToDouble(bytes, index);
var v = BitConverter.Int64BitsToDouble(BinaryPrimitives.ReadInt64LittleEndian(bytes.AsSpan(index)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we can just work with span instead of bytes in all cases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve updated the code to work directly with Span<byte> instead of using byte[] where applicable, as you suggested. Please verify and let me know if any further changes are needed.

@medhatiwari medhatiwari changed the title Fix deserialization of primitive arrays on Big Endian systems CSHARP-5614: Fix deserialization of primitive arrays on Big Endian systems Jun 11, 2025
@medhatiwari
Copy link
Contributor Author

Looks good overall. Few asks/questions:

  1. Could you please file a separate Jira ticket for each PR, describing the issue? And also update the PR name according to our conventions (see the closed PRs for this).
  2. Did you observe any failing failing tests without this fix?

Hi

So I have opened the JIra ticket

and I was getting test failures in MongoDB.Bson Test bucket, I have listed all of them below

[xUnit.net 00:00:14.98]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_special_values<Double>(array: [1.7976931348623157E+308, -1.7976931348623157E+308, 4.9406564584124654E-324, -∞, ∞, ...]) [FAIL]
[xUnit.net 00:00:15.00]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_special_values<Int64>(array: [9223372036854775807, -9223372036854775808]) [FAIL]
[xUnit.net 00:00:15.01]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_special_values<Decimal>(array: [79228162514264337593543950335, -79228162514264337593543950335, 1, 0, -1]) [FAIL]
[xUnit.net 00:00:15.01]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_special_values<SByte>(array: [127, -128]) [FAIL]
[xUnit.net 00:00:15.02]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_special_values<UInt16>(array: [65535, 0]) [FAIL]
[xUnit.net 00:00:15.04]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_special_values<Single>(array: [3.40282347E+38, -3.40282347E+38, 1.40129846E-45, -∞, ∞, ...]) [FAIL]
[xUnit.net 00:00:15.05]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_special_values<Int16>(array: [32767, -32768]) [FAIL]
[xUnit.net 00:00:15.06]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_special_values<Int32>(array: [2147483647, -2147483648]) [FAIL]
[xUnit.net 00:00:15.13]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<UInt64>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.26]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<Single>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.31]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<Char>(array: ['\0', 0x0001, 0x0002, 0x0003, 0x0004, ...]) [FAIL]
[xUnit.net 00:00:15.37]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<UInt32>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.41]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<SByte>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.47]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<Decimal>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.51]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<UInt16>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.55]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<Int32>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.69]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<Double>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.74]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<Int16>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.79]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.ReadonlyMemory_should_roundtrip_equivalent_to_array<Int64>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:15.79]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values<Int64>(array: [9223372036854775807, -9223372036854775808]) [FAIL]
[xUnit.net 00:00:15.79]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values<Decimal>(array: [79228162514264337593543950335, -79228162514264337593543950335, 1, 0, -1]) [FAIL]
[xUnit.net 00:00:15.81]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values<Single>(array: [3.40282347E+38, -3.40282347E+38, 1.40129846E-45, -∞, ∞, ...]) [FAIL]
[xUnit.net 00:00:15.82]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values<SByte>(array: [127, -128]) [FAIL]
[xUnit.net 00:00:15.82]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values<UInt16>(array: [65535, 0]) [FAIL]
[xUnit.net 00:00:15.84]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values<Double>(array: [1.7976931348623157E+308, -1.7976931348623157E+308, 4.9406564584124654E-324, -∞, ∞, ...]) [FAIL]
[xUnit.net 00:00:15.84]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values<Int32>(array: [2147483647, -2147483648]) [FAIL]
[xUnit.net 00:00:15.84]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values<Int16>(array: [32767, -32768]) [FAIL]
[xUnit.net 00:00:15.98]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<Single>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.03]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<UInt64>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.08]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<Int32>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.12]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<Int16>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.16]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<UInt16>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.20]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<SByte>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.25]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<Char>(array: ['\0', 0x0001, 0x0002, 0x0003, 0x0004, ...]) [FAIL]
[xUnit.net 00:00:16.30]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<Decimal>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.44]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<Double>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.48]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<Int64>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.52]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_equivalent_to_array<UInt32>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.56]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<Int32>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.67]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<Single>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.70]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<UInt32>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.72]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<Int16>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.75]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<UInt16>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.77]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<SByte>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.79]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<Decimal>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.90]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<Double>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.94]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<Char>(array: ['\0', 0x0001, 0x0002, 0x0003, 0x0004, ...]) [FAIL]
[xUnit.net 00:00:16.96]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<UInt64>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:16.99]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Memory_should_roundtrip_special_values_correctly<Int64>(array: [0, 1, 2, 3, 4, ...]) [FAIL]
[xUnit.net 00:00:17.00]     MongoDB.Bson.Tests.Serialization.MemorySerializerTests.Mixed_memory_types_should_roundtrip_correctly [FAIL]

@medhatiwari medhatiwari requested a review from BorisDog June 11, 2025 18:52
BorisDog
BorisDog requested changes Jun 11, 2025
View reviewed changes
@medhatiwari
CSHARP-5614: added BinaryPrimitivesCompat.ReadDoubleLittleEndian for …
31d88d5 
…d2d conversion and reverted extra spaces

Signed-off-by: Medha Tiwari <[email protected]>
@medhatiwari medhatiwari requested a review from BorisDog June 11, 2025 20:09
@BorisDog BorisDog merged commit 5d4827a into mongodb:main Jun 12, 2025
27 of 31 checks passed
@BorisDog
Copy link
Contributor

Thanks @medhatiwari!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BorisDog BorisDog Awaiting requested review from BorisDog

Assignees
No one assigned
Labels
improvement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@medhatiwari @rstam @BorisDog